Implement "enahanced telemetry reporting" #7360

andrewlock · 2025-08-07T12:41:36Z

Summary of changes

Implements "enhance telemetry reporting" for .NET

Reason for change

We want to report the telemetry found from all sources, to implement this RFC, so that customers can more easily understand where their telemetry comes from.

Implementation details

The .NET implementation was already pretty close to the desired result, so there were just some main changes to make

Read sources in lowest to highest priority, and always report all found values.
- This differs from previously, where we would read from highest to lowest and stop on first found values.
- In order to ensure that the "last" reported telemetry value is the actual value we use, we have to record the telemetry value twice in cases where we have a parse failure (which should be rare in practice).
- To do this properly, we had to explicitly expose the Origin on IConfigurationSource and captuate the "value for telemetry" where required (for non-simple types).
Report the "default" value to be used, even if it's going to be overridden by an explicit value.
- This is in keeping with the general "report all sources" principle.
- We have to do the same "fixing" of values when we have parsing errors as above.
- To avoid expensive unnecessary computation where a default value is not "simple" we go against this approach when a Func<T> is provided for calculating the default.
For "fallback" keys, we just treat them the same as we do today. The backend will handle merging the keys

Test coverage

Mostly covered by existing tests, and added/updated unit tests to meet new expectations

Other details

Spec: https://docs.google.com/document/d/1vhIimn2vt4tDRSxsHn6vWSc8zYHl0Lv0Fk7CQps04C4/edit?tab=t.0#heading=h.7tw832mj0ztp

Dependent (stacked) on

dd-trace-dotnet-ci-bot · 2025-08-07T13:51:36Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7360) - mean (72ms)  : 71, 73
     .   : milestone, 72,
    master - mean (72ms)  : 71, 73
     .   : milestone, 72,

    section Baseline
    This PR (7360) - mean (68ms)  : 65, 72
     .   : milestone, 68,
    master - mean (68ms)  : 67, 70
     .   : milestone, 68,

    section CallTarget+Inlining+NGEN
    This PR (7360) - mean (996ms)  : 979, 1012
     .   : milestone, 996,
    master - mean (1,026ms)  : 1003, 1049
     .   : milestone, 1026,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7360) - mean (107ms)  : 106, 108
     .   : milestone, 107,
    master - mean (107ms)  : 106, 109
     .   : milestone, 107,

    section Baseline
    This PR (7360) - mean (106ms)  : 104, 108
     .   : milestone, 106,
    master - mean (107ms)  : 104, 109
     .   : milestone, 107,

    section CallTarget+Inlining+NGEN
    This PR (7360) - mean (710ms)  : 703, 718
     .   : milestone, 710,
    master - mean (726ms)  : 717, 734
     .   : milestone, 726,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7360) - mean (94ms)  : 93, 95
     .   : milestone, 94,
    master - mean (94ms)  : 93, 95
     .   : milestone, 94,

    section Baseline
    This PR (7360) - mean (93ms)  : 91, 95
     .   : milestone, 93,
    master - mean (94ms)  : 91, 96
     .   : milestone, 94,

    section CallTarget+Inlining+NGEN
    This PR (7360) - mean (666ms)  : 656, 677
     .   : milestone, 666,
    master - mean (681ms)  : 671, 691
     .   : milestone, 681,

gantt
    title Execution time (ms) FakeDbCommand (.NET 8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7360) - mean (93ms)  : 91, 94
     .   : milestone, 93,
    master - mean (93ms)  : 92, 94
     .   : milestone, 93,

    section Baseline
    This PR (7360) - mean (92ms)  : 90, 94
     .   : milestone, 92,
    master - mean (92ms)  : 90, 94
     .   : milestone, 92,

    section CallTarget+Inlining+NGEN
    This PR (7360) - mean (601ms)  : 592, 610
     .   : milestone, 601,
    master - mean (624ms)  : 614, 634
     .   : milestone, 624,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7360) - mean (199ms)  : 196, 202
     .   : milestone, 199,
    master - mean (194ms)  : 191, 196
     .   : milestone, 194,

    section Baseline
    This PR (7360) - mean (196ms)  : 191, 201
     .   : milestone, 196,
    master - mean (190ms)  : 186, 195
     .   : milestone, 190,

    section CallTarget+Inlining+NGEN
    This PR (7360) - mean (1,129ms)  : 1102, 1155
     .   : milestone, 1129,
    master - mean (1,136ms)  : 1109, 1164
     .   : milestone, 1136,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7360) - mean (284ms)  : 277, 292
     .   : milestone, 284,
    master - mean (278ms)  : 271, 285
     .   : milestone, 278,

    section Baseline
    This PR (7360) - mean (283ms)  : 276, 290
     .   : milestone, 283,
    master - mean (275ms)  : 270, 280
     .   : milestone, 275,

    section CallTarget+Inlining+NGEN
    This PR (7360) - mean (921ms)  : 898, 944
     .   : milestone, 921,
    master - mean (913ms)  : 887, 940
     .   : milestone, 913,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7360) - mean (277ms)  : 271, 283
     .   : milestone, 277,
    master - mean (268ms)  : 265, 272
     .   : milestone, 268,

    section Baseline
    This PR (7360) - mean (277ms)  : 270, 283
     .   : milestone, 277,
    master - mean (268ms)  : 263, 273
     .   : milestone, 268,

    section CallTarget+Inlining+NGEN
    This PR (7360) - mean (909ms)  : 878, 939
     .   : milestone, 909,
    master - mean (899ms)  : 880, 917
     .   : milestone, 899,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7360) - mean (279ms)  : 274, 284
     .   : milestone, 279,
    master - mean (267ms)  : 264, 269
     .   : milestone, 267,

    section Baseline
    This PR (7360) - mean (279ms)  : 272, 286
     .   : milestone, 279,
    master - mean (267ms)  : 263, 270
     .   : milestone, 267,

    section CallTarget+Inlining+NGEN
    This PR (7360) - mean (822ms)  : 800, 844
     .   : milestone, 822,
    master - mean (816ms)  : 795, 836
     .   : milestone, 816,

daniel-romano-DD · 2025-08-11T14:47:11Z

tracer/src/Datadog.Trace/Configuration/ConfigurationSources/CompositeConfigurationSource.cs

-            => _sources
-              .Select(source => source.GetString(key, telemetry, validator, recordValue))
-              .FirstOrDefault(value => value.IsValid, ConfigurationResult<string>.NotFound());
+        {


The only difference in all these GetXXX is the value retrieving function, which could be a lambda, to avoid the n-plicated code

Yeah, I've gone back and forth on the lambda thing. I extracted these to lambdas in ConfigurationBuilder and it makes everything seem that much more complicated, that I'm really not sure which is best. Add to that the fact that there's slightly different behaviour between each of the cases and it ends up really not being clear-cut which approach is better tbh.

daniel-romano-DD

Nice work

pr-commenter · 2025-08-11T18:10:16Z

Benchmarks

Benchmarks Report for benchmark platform 🐌

Benchmarks for #7360 compared to master:

1 benchmarks are faster, with geometric mean 1.163
9 benchmarks have fewer allocations
7 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #7360

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.ActivityBenchmark.StartStopWithChild‑net6.0	5.52 KB	5.62 KB	102 B	1.85%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	10.3μs	56.9ns	327ns	0	0	0	5.52 KB
master	`StartStopWithChild`	netcoreapp3.1	13.3μs	66.6ns	305ns	0	0	0	5.68 KB
master	`StartStopWithChild`	net472	21.9μs	111ns	588ns	0.858	0.214	0	6.04 KB
#7360	`StartStopWithChild`	net6.0	10.6μs	59.7ns	409ns	0	0	0	5.62 KB
#7360	`StartStopWithChild`	netcoreapp3.1	13.4μs	65.8ns	279ns	0	0	0	5.69 KB
#7360	`StartStopWithChild`	net472	22μs	124ns	861ns	0.977	0.326	0.109	6.03 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	936μs	62.9ns	235ns	2.71 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	1.05ms	1.77μs	6.4μs	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	1.2ms	56.4ns	218ns	3.31 KB
#7360	`WriteAndFlushEnrichedTraces`	net6.0	917μs	130ns	488ns	2.71 KB
#7360	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	1.02ms	115ns	446ns	2.71 KB
#7360	`WriteAndFlushEnrichedTraces`	net472	1.22ms	279ns	1.08μs	3.31 KB

Benchmarks.Trace.Asm.AppSecBodyBenchmark - Same speed ✔️ Fewer allocations 🎉

Fewer allocations 🎉 in #7360

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody‑net6.0	182.92 KB	176.95 KB	-5.97 KB	-3.26%
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody‑net6.0	179.42 KB	173.45 KB	-5.97 KB	-3.33%
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody‑netcoreapp3.1	186.66 KB	179.06 KB	-7.6 KB	-4.07%
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody‑netcoreapp3.1	183.24 KB	175.64 KB	-7.6 KB	-4.15%
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody‑net472	213.49 KB	199.31 KB	-14.18 KB	-6.64%
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody‑net472	209.98 KB	195.79 KB	-14.18 KB	-6.75%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`AllCycleSimpleBody`	net6.0	323μs	903ns	3.5μs	0	0	179.42 KB
master	`AllCycleSimpleBody`	netcoreapp3.1	496μs	2.64μs	13.7μs	0	0	183.24 KB
master	`AllCycleSimpleBody`	net472	422μs	97.9ns	353ns	33.3	2.08	209.98 KB
master	`AllCycleMoreComplexBody`	net6.0	333μs	204ns	762ns	0	0	182.92 KB
master	`AllCycleMoreComplexBody`	netcoreapp3.1	516μs	198ns	742ns	0	0	186.66 KB
master	`AllCycleMoreComplexBody`	net472	433μs	155ns	600ns	32.3	0	213.49 KB
master	`ObjectExtractorSimpleBody`	net6.0	320ns	1.57ns	6.83ns	0	0	280 B
master	`ObjectExtractorSimpleBody`	netcoreapp3.1	399ns	1.43ns	5.33ns	0	0	272 B
master	`ObjectExtractorSimpleBody`	net472	299ns	0.0467ns	0.175ns	0.0444	0	281 B
master	`ObjectExtractorMoreComplexBody`	net6.0	6.28μs	3.9ns	15.1ns	0	0	3.78 KB
master	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	7.74μs	34.4ns	129ns	0	0	3.69 KB
master	`ObjectExtractorMoreComplexBody`	net472	6.68μs	1.82ns	6.8ns	0.601	0	3.8 KB
#7360	`AllCycleSimpleBody`	net6.0	317μs	1.32μs	5.12μs	0	0	173.45 KB
#7360	`AllCycleSimpleBody`	netcoreapp3.1	489μs	1.67μs	6.46μs	0	0	175.64 KB
#7360	`AllCycleSimpleBody`	net472	424μs	120ns	463ns	29.2	0	195.79 KB
#7360	`AllCycleMoreComplexBody`	net6.0	330μs	546ns	2.12μs	0	0	176.95 KB
#7360	`AllCycleMoreComplexBody`	netcoreapp3.1	487μs	696ns	2.7μs	0	0	179.06 KB
#7360	`AllCycleMoreComplexBody`	net472	437μs	69.6ns	270ns	30.2	0	199.31 KB
#7360	`ObjectExtractorSimpleBody`	net6.0	326ns	0.204ns	0.79ns	0	0	280 B
#7360	`ObjectExtractorSimpleBody`	netcoreapp3.1	402ns	2.17ns	11.3ns	0	0	272 B
#7360	`ObjectExtractorSimpleBody`	net472	303ns	0.0409ns	0.147ns	0.0443	0	281 B
#7360	`ObjectExtractorMoreComplexBody`	net6.0	6.36μs	4.37ns	16.9ns	0	0	3.78 KB
#7360	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	7.77μs	36.5ns	141ns	0	0	3.69 KB
#7360	`ObjectExtractorMoreComplexBody`	net472	6.68μs	4.44ns	17.2ns	0.599	0	3.8 KB

Benchmarks.Trace.Asm.AppSecEncoderBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EncodeArgs`	net6.0	77.5μs	80.9ns	280ns	0	32.4 KB
master	`EncodeArgs`	netcoreapp3.1	98.4μs	73ns	283ns	0	32.4 KB
master	`EncodeArgs`	net472	113μs	14.7ns	56.9ns	5.08	32.51 KB
master	`EncodeLegacyArgs`	net6.0	140μs	15.1ns	58.6ns	0	2.15 KB
master	`EncodeLegacyArgs`	netcoreapp3.1	199μs	215ns	831ns	0	2.14 KB
master	`EncodeLegacyArgs`	net472	262μs	29.3ns	113ns	0	2.16 KB
#7360	`EncodeArgs`	net6.0	78μs	43.8ns	170ns	0	32.4 KB
#7360	`EncodeArgs`	netcoreapp3.1	96.5μs	216ns	835ns	0	32.4 KB
#7360	`EncodeArgs`	net472	110μs	16.7ns	62.6ns	4.95	32.51 KB
#7360	`EncodeLegacyArgs`	net6.0	148μs	16.2ns	60.7ns	0	2.15 KB
#7360	`EncodeLegacyArgs`	netcoreapp3.1	196μs	139ns	481ns	0	2.14 KB
#7360	`EncodeLegacyArgs`	net472	261μs	34.3ns	128ns	0	2.16 KB

Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Allocated
master	`RunWafRealisticBenchmark`	net6.0	404μs	87.2ns	315ns	4.55 KB
master	`RunWafRealisticBenchmark`	netcoreapp3.1	412μs	77.2ns	289ns	4.48 KB
master	`RunWafRealisticBenchmark`	net472	449μs	85.6ns	332ns	4.66 KB
master	`RunWafRealisticBenchmarkWithAttack`	net6.0	288μs	207ns	800ns	2.24 KB
master	`RunWafRealisticBenchmarkWithAttack`	netcoreapp3.1	302μs	79.3ns	286ns	2.22 KB
master	`RunWafRealisticBenchmarkWithAttack`	net472	316μs	26.7ns	103ns	2.29 KB
#7360	`RunWafRealisticBenchmark`	net6.0	394μs	100ns	389ns	4.55 KB
#7360	`RunWafRealisticBenchmark`	netcoreapp3.1	417μs	171ns	615ns	4.48 KB
#7360	`RunWafRealisticBenchmark`	net472	432μs	57.9ns	224ns	4.66 KB
#7360	`RunWafRealisticBenchmarkWithAttack`	net6.0	292μs	85.8ns	332ns	2.24 KB
#7360	`RunWafRealisticBenchmarkWithAttack`	netcoreapp3.1	301μs	99.9ns	360ns	2.22 KB
#7360	`RunWafRealisticBenchmarkWithAttack`	net472	311μs	24.8ns	92.8ns	2.29 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Allocated
master	`SendRequest`	net6.0	61.7μs	28ns	105ns	14.52 KB
master	`SendRequest`	netcoreapp3.1	71.3μs	64.9ns	252ns	17.42 KB
master	`SendRequest`	net472	0.016ns	0.00187ns	0.00725ns	0 b
#7360	`SendRequest`	net6.0	59.6μs	79.7ns	287ns	14.52 KB
#7360	`SendRequest`	netcoreapp3.1	70.4μs	209ns	808ns	17.42 KB
#7360	`SendRequest`	net472	0.0154ns	0.00188ns	0.00728ns	0 b

Benchmarks.Trace.CharSliceBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #7360

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice‑net472	0 b	73 B	73 B	∞
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool‑net472	0 b	46 B	46 B	∞
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool‑net6.0	3 B	4 B	1 B	33.33%

Fewer allocations 🎉 in #7360

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice‑net6.0	4 B	2 B	-2 B	-50.00%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`OriginalCharSlice`	net6.0	1.87ms	8.54μs	32μs	0	640 KB
master	`OriginalCharSlice`	netcoreapp3.1	2.16ms	5.45μs	21.1μs	0	640 KB
master	`OriginalCharSlice`	net472	2.69ms	551ns	2.14μs	100	641.95 KB
master	`OptimizedCharSlice`	net6.0	1.41ms	78.9ns	295ns	0	4 B
master	`OptimizedCharSlice`	netcoreapp3.1	1.7ms	215ns	834ns	0	1 B
master	`OptimizedCharSlice`	net472	2.04ms	657ns	2.55μs	0	0 b
master	`OptimizedCharSliceWithPool`	net6.0	800μs	132ns	513ns	0	3 B
master	`OptimizedCharSliceWithPool`	netcoreapp3.1	810μs	63.4ns	245ns	0	1 B
master	`OptimizedCharSliceWithPool`	net472	1.13ms	49.2ns	191ns	0	0 b
#7360	`OriginalCharSlice`	net6.0	1.88ms	541ns	2.09μs	0	640 KB
#7360	`OriginalCharSlice`	netcoreapp3.1	2.18ms	4.59μs	17.2μs	0	640 KB
#7360	`OriginalCharSlice`	net472	2.62ms	122ns	441ns	100	641.95 KB
#7360	`OptimizedCharSlice`	net6.0	1.4ms	590ns	2.28μs	0	2 B
#7360	`OptimizedCharSlice`	netcoreapp3.1	1.73ms	136ns	527ns	0	1 B
#7360	`OptimizedCharSlice`	net472	2.02ms	383ns	1.48μs	0	73 B
#7360	`OptimizedCharSliceWithPool`	net6.0	804μs	27.8ns	108ns	0	4 B
#7360	`OptimizedCharSliceWithPool`	netcoreapp3.1	810μs	199ns	770ns	0	1 B
#7360	`OptimizedCharSliceWithPool`	net472	1.13ms	42.4ns	164ns	0	46 B

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Faster 🎉 More allocations ⚠️

Faster 🎉 in #7360

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑netcoreapp3.1	1.163	760,256.25	653,742.19	several?

More allocations ⚠️ in #7360

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑net6.0	41.68 KB	42.65 KB	975 B	2.34%

Fewer allocations 🎉 in #7360

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑netcoreapp3.1	42.13 KB	41.89 KB	-237 B	-0.56%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	636μs	429ns	1.55μs	0	41.68 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	734μs	5.06μs	48.3μs	0	42.13 KB
master	`WriteAndFlushEnrichedTraces`	net472	935μs	3.26μs	12.6μs	8.33	56.22 KB
#7360	`WriteAndFlushEnrichedTraces`	net6.0	633μs	3.25μs	23.2μs	0	42.65 KB
#7360	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	659μs	2.86μs	11.1μs	0	41.89 KB
#7360	`WriteAndFlushEnrichedTraces`	net472	985μs	2.79μs	10.8μs	4.81	55.98 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteNonQuery`	net6.0	1.94μs	1.73ns	6.68ns	0	0	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	2.47μs	12.9ns	68.4ns	0	0	1.02 KB
master	`ExecuteNonQuery`	net472	2.7μs	4.29ns	14.9ns	0.149	0.0136	987 B
#7360	`ExecuteNonQuery`	net6.0	1.91μs	0.699ns	2.52ns	0	0	1.02 KB
#7360	`ExecuteNonQuery`	netcoreapp3.1	2.49μs	9.43ns	36.5ns	0	0	1.02 KB
#7360	`ExecuteNonQuery`	net472	2.66μs	3.01ns	11.6ns	0.149	0.0135	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.75μs	4.73ns	18.3ns	0	1.03 KB
master	`CallElasticsearch`	netcoreapp3.1	2.31μs	9.02ns	33.7ns	0	1.03 KB
master	`CallElasticsearch`	net472	3.57μs	2.02ns	7.84ns	0.162	1.04 KB
master	`CallElasticsearchAsync`	net6.0	1.83μs	5.8ns	21.7ns	0	1.01 KB
master	`CallElasticsearchAsync`	netcoreapp3.1	2.41μs	7.56ns	27.3ns	0	1.08 KB
master	`CallElasticsearchAsync`	net472	3.7μs	2.3ns	8.91ns	0.166	1.1 KB
#7360	`CallElasticsearch`	net6.0	1.78μs	1.7ns	6.6ns	0	1.03 KB
#7360	`CallElasticsearch`	netcoreapp3.1	2.35μs	11.1ns	43.1ns	0	1.03 KB
#7360	`CallElasticsearch`	net472	3.66μs	1.86ns	6.96ns	0.165	1.04 KB
#7360	`CallElasticsearchAsync`	net6.0	1.8μs	5.59ns	20.9ns	0	1.01 KB
#7360	`CallElasticsearchAsync`	netcoreapp3.1	2.42μs	8.74ns	33.8ns	0	1.08 KB
#7360	`CallElasticsearchAsync`	net472	3.9μs	2.56ns	9.59ns	0.174	1.1 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.79μs	7.44ns	28.8ns	0	952 B
master	`ExecuteAsync`	netcoreapp3.1	2.29μs	6.97ns	27ns	0	952 B
master	`ExecuteAsync`	net472	2.54μs	4.48ns	17.3ns	0.139	915 B
#7360	`ExecuteAsync`	net6.0	1.85μs	4.46ns	17.3ns	0	952 B
#7360	`ExecuteAsync`	netcoreapp3.1	2.43μs	9.16ns	34.3ns	0	952 B
#7360	`ExecuteAsync`	net472	2.5μs	1.32ns	5.11ns	0.137	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	7.18μs	27.2ns	98.2ns	0	2.36 KB
master	`SendAsync`	netcoreapp3.1	8.37μs	12.4ns	47.9ns	0	2.9 KB
master	`SendAsync`	net472	12.1μs	14.8ns	57.4ns	0.483	3.18 KB
#7360	`SendAsync`	net6.0	7.03μs	12ns	46.3ns	0	2.36 KB
#7360	`SendAsync`	netcoreapp3.1	8.53μs	29.5ns	114ns	0	2.9 KB
#7360	`SendAsync`	net472	12.4μs	9.22ns	35.7ns	0.495	3.18 KB

Benchmarks.Trace.Iast.StringAspectsBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #7360

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑netcoreapp3.1	259.73 KB	277.7 KB	17.97 KB	6.92%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑netcoreapp3.1	42.64 KB	44.04 KB	1.4 KB	3.28%

Fewer allocations 🎉 in #7360

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑net6.0	43.82 KB	43.13 KB	-696 B	-1.59%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Allocated
master	`StringConcatBenchmark`	net6.0	44.4μs	230ns	1.17μs	43.82 KB
master	`StringConcatBenchmark`	netcoreapp3.1	47.7μs	273ns	1.85μs	42.64 KB
master	`StringConcatBenchmark`	net472	57.2μs	148ns	574ns	57.34 KB
master	`StringConcatAspectBenchmark`	net6.0	474μs	2.06μs	7.15μs	276.89 KB
master	`StringConcatAspectBenchmark`	netcoreapp3.1	487μs	2.15μs	8.06μs	259.73 KB
master	`StringConcatAspectBenchmark`	net472	400μs	2.29μs	17.1μs	278.53 KB
#7360	`StringConcatBenchmark`	net6.0	45.7μs	264ns	1.89μs	43.13 KB
#7360	`StringConcatBenchmark`	netcoreapp3.1	46.5μs	215ns	774ns	44.04 KB
#7360	`StringConcatBenchmark`	net472	56.7μs	192ns	719ns	57.34 KB
#7360	`StringConcatAspectBenchmark`	net6.0	504μs	2.29μs	8.57μs	277.6 KB
#7360	`StringConcatAspectBenchmark`	netcoreapp3.1	512μs	2.14μs	7.72μs	277.7 KB
#7360	`StringConcatAspectBenchmark`	net472	405μs	2.2μs	12.6μs	278.3 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.54μs	2.98ns	11.5ns	0	1.7 KB
master	`EnrichedLog`	netcoreapp3.1	3.49μs	12.1ns	47ns	0	1.7 KB
master	`EnrichedLog`	net472	3.88μs	4.71ns	17.6ns	0.252	1.64 KB
#7360	`EnrichedLog`	net6.0	2.57μs	4.96ns	19.2ns	0	1.7 KB
#7360	`EnrichedLog`	netcoreapp3.1	3.38μs	17ns	72.2ns	0	1.7 KB
#7360	`EnrichedLog`	net472	4.04μs	7.16ns	27.7ns	0.241	1.64 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Allocated
master	`EnrichedLog`	net6.0	123μs	24.5ns	94.9ns	4.31 KB
master	`EnrichedLog`	netcoreapp3.1	126μs	64.1ns	231ns	4.31 KB
master	`EnrichedLog`	net472	166μs	44.1ns	171ns	4.52 KB
#7360	`EnrichedLog`	net6.0	121μs	61.9ns	232ns	4.31 KB
#7360	`EnrichedLog`	netcoreapp3.1	128μs	368ns	1.43μs	4.31 KB
#7360	`EnrichedLog`	net472	165μs	41.6ns	161ns	4.51 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	4.9μs	6.72ns	25.2ns	0	2.26 KB
master	`EnrichedLog`	netcoreapp3.1	6.52μs	17.6ns	63.3ns	0	2.26 KB
master	`EnrichedLog`	net472	7.58μs	5.79ns	21.7ns	0.304	2.08 KB
#7360	`EnrichedLog`	net6.0	4.99μs	20.6ns	79.9ns	0	2.26 KB
#7360	`EnrichedLog`	netcoreapp3.1	6.68μs	17.5ns	60.5ns	0	2.26 KB
#7360	`EnrichedLog`	net472	7.43μs	6.87ns	25.7ns	0.298	2.08 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.98μs	1.84ns	7.13ns	0	1.2 KB
master	`SendReceive`	netcoreapp3.1	2.59μs	5.82ns	22.5ns	0	1.2 KB
master	`SendReceive`	net472	3.26μs	5.66ns	21.9ns	0.177	1.2 KB
#7360	`SendReceive`	net6.0	1.98μs	9.5ns	38ns	0	1.2 KB
#7360	`SendReceive`	netcoreapp3.1	2.66μs	12.4ns	49.5ns	0	1.2 KB
#7360	`SendReceive`	net472	3.19μs	3.04ns	11.8ns	0.176	1.2 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	4.2μs	4.39ns	16.4ns	0	1.58 KB
master	`EnrichedLog`	netcoreapp3.1	5.56μs	16.2ns	58.4ns	0	1.63 KB
master	`EnrichedLog`	net472	6.44μs	6.69ns	24.1ns	0.322	2.03 KB
#7360	`EnrichedLog`	net6.0	4.06μs	16.8ns	65.3ns	0	1.58 KB
#7360	`EnrichedLog`	netcoreapp3.1	5.49μs	7.67ns	29.7ns	0	1.63 KB
#7360	`EnrichedLog`	net472	6.49μs	6.82ns	24.6ns	0.291	2.03 KB

Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	740ns	3.45ns	15.8ns	0	576 B
master	`StartFinishSpan`	netcoreapp3.1	931ns	4.9ns	21.9ns	0	576 B
master	`StartFinishSpan`	net472	905ns	0.276ns	1.07ns	0.0914	578 B
master	`StartFinishScope`	net6.0	870ns	0.3ns	1.08ns	0	696 B
master	`StartFinishScope`	netcoreapp3.1	1.16μs	6.29ns	33.3ns	0	696 B
master	`StartFinishScope`	net472	1.14μs	0.0495ns	0.171ns	0.103	658 B
#7360	`StartFinishSpan`	net6.0	744ns	3.8ns	16.6ns	0	576 B
#7360	`StartFinishSpan`	netcoreapp3.1	931ns	4.29ns	17.2ns	0	576 B
#7360	`StartFinishSpan`	net472	905ns	0.404ns	1.46ns	0.0907	578 B
#7360	`StartFinishScope`	net6.0	898ns	0.375ns	1.45ns	0	696 B
#7360	`StartFinishScope`	netcoreapp3.1	1.11μs	5.25ns	21ns	0	696 B
#7360	`StartFinishScope`	net472	1.13μs	1.13ns	4.24ns	0.102	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	1.03μs	5.53ns	28.2ns	0	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	1.37μs	6.88ns	32.3ns	0	696 B
master	`RunOnMethodBegin`	net472	1.39μs	1.99ns	7.7ns	0.0979	658 B
#7360	`RunOnMethodBegin`	net6.0	996ns	5.19ns	24.4ns	0	696 B
#7360	`RunOnMethodBegin`	netcoreapp3.1	1.41μs	6.6ns	24.7ns	0	696 B
#7360	`RunOnMethodBegin`	net472	1.37μs	0.266ns	1.03ns	0.103	658 B

This is necessary so that we can "correct" telemetry that is overridden in the CompositeConfigurationSource

…ig for all of them

andrewlock marked this pull request as ready for review August 7, 2025 14:01

andrewlock requested a review from a team as a code owner August 7, 2025 14:01

daniel-romano-DD reviewed Aug 11, 2025

View reviewed changes

daniel-romano-DD approved these changes Aug 11, 2025

View reviewed changes

andrewlock requested a review from a team as a code owner August 11, 2025 17:09

link04 approved these changes Aug 11, 2025

View reviewed changes

Base automatically changed from andrew/more-telemetry-refactor to master August 12, 2025 14:24

andrewlock requested a review from a team as a code owner August 12, 2025 14:24

andrewlock force-pushed the andrew/config-chaining branch 2 times, most recently from bb84c83 to d004707 Compare August 13, 2025 15:00

andrewlock added 7 commits August 14, 2025 13:20

Expose Origin on IConfigurationSource

e9b74cd

Add "telemetry override" parameter to ConfigurationResult.

4280755

This is necessary so that we can "correct" telemetry that is overridden in the CompositeConfigurationSource

Search the configuration sources in reverse, to ensure we record conf…

7452c7c

…ig for all of them

"Fix" ordering of recorded telemetry

eca6866

Fix incorrect use of lambda in bool accessor

99bebe3

Add additional integration test

9cb76fc

Fix rebasing issue

0ce0759

andrewlock force-pushed the andrew/config-chaining branch from d004707 to 0ce0759 Compare August 14, 2025 12:20

andrewlock merged commit 7b8127e into master Aug 14, 2025
150 of 151 checks passed

andrewlock deleted the andrew/config-chaining branch August 14, 2025 16:54

github-actions bot added this to the vNext-v3 milestone Aug 14, 2025

andrewlock added area:tracer The core tracer library (Datadog.Trace, does not include OpenTracing, native code, or integrations) type:new-feature area:telemetry labels Aug 26, 2025

Implement "enahanced telemetry reporting" #7360

Implement "enahanced telemetry reporting" #7360

Uh oh!

Conversation

andrewlock commented Aug 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

Uh oh!

dd-trace-dotnet-ci-bot bot commented Aug 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Execution-Time Benchmarks Report ⏱️

Uh oh!

daniel-romano-DD Aug 11, 2025

Choose a reason for hiding this comment

Uh oh!

andrewlock Aug 12, 2025

Choose a reason for hiding this comment

Uh oh!

daniel-romano-DD left a comment

Choose a reason for hiding this comment

Uh oh!

pr-commenter bot commented Aug 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Benchmarks Report for benchmark platform 🐌

Benchmark details

More allocations ⚠️ in #7360

Raw results

Raw results

Fewer allocations 🎉 in #7360

Raw results

Raw results

Raw results

Raw results

More allocations ⚠️ in #7360

Fewer allocations 🎉 in #7360

Raw results

Faster 🎉 in #7360

More allocations ⚠️ in #7360

Fewer allocations 🎉 in #7360

Raw results

Raw results

Raw results

Raw results

Raw results

More allocations ⚠️ in #7360

Fewer allocations 🎉 in #7360

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Uh oh!

Uh oh!

Uh oh!

andrewlock commented Aug 7, 2025 •

edited

Loading

dd-trace-dotnet-ci-bot bot commented Aug 7, 2025 •

edited

Loading

pr-commenter bot commented Aug 11, 2025 •

edited

Loading